Picture encoding method and apparatus and picture decoding method and apparatus

ABSTRACT

A picture encoding method includes receiving an input video signal, encoding the video signal using a reference picture signal to generate a video code stream, encoding the reference picture signal to generate a reference picture code stream, and multiplexing the video code stream with the reference picture code stream to generate an output code stream.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present divisional application claims the benefit of priority under35 U.S.C. §120 to application Ser. No. 10/661,697, filed Sep. 15, 2003,which is a Continuation Application of PCT Application No.PCT/JP03/00426, filed Jan. 20, 2003, which was not published under PCTArticle 21(2) in English; and under 35 U.S.C. § 119 from Japaneseapplications Nos. 2002-010875, filed on Jan. 18, 2002, and 2003-010135,filed on Jan. 17, 2003, the entire contents of each are herebyincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a picture encoding method ofcompression-encoding a picture in a few number of bits and a picturedecoding method of playing back a picture by decoding a code streamobtained by compression encoding and, more particularly, to a pictureencoding method and apparatus and a picture decoding method andapparatus which can make a recovery from the adverse effect of an erroras fast as possible without degrading the encoding efficiency whentransmitting/storing encoded data through a transmission pathsusceptible to errors.

2. Description of the Related Art

It is necessary to compression-encode pictures in a few number of bitsin order to transmit or store the pictures in systems designed totransmit or store pictures, e.g., a videophone, video conference system,portable information terminal, digital video disk system, and digital TVbroadcasting system.

As such compression encoding techniques, various schemes have beendeveloped, including a motion compensation scheme, discrete cosinetransform scheme, subband encoding scheme, pyramid encoding scheme, andcombinations thereof. The following are defined as internationalstandards for video compression encoding: ISO•MPEG-1, MPEG-2, MPEG-4,ITU-T H.261, H.262, H.263, and the like.

All these schemes are compression encoding schemes based on acombination of motion compensation adaptive prediction and discretecosine transform, which are described in detail in reference 1 (HiroshiYasuda, “MPEG/International Standardization of Multimedia Encoding”,Maruzen) and the like.

A conventional picture encoding/decoding apparatus has the followingproblems. First, in a communication path with the possibility of beingmixed with errors, such as a radio communication path, performing onlyencoding will lead to considerable deterioration in decoded picturequality upon occurrence of an error. When errors occur in signals suchas a sync signal, mode information, and motion vector information, inparticular, picture quality noticeably deteriorates.

Second, in motion compensation adaptive predictive encoding used forpicture encoding, only the difference between frames is encoded. Forthis reason, if an error occurs, the corresponding frame fails, and anerroneous picture is stored in a frame memory. A predictive picture isgenerated by using the erroneous picture, and the residual error isadded to the predictive error. As a consequence, even if subsequentframes are properly decoded, proper pictures cannot be obtained from thesubsequent frames unless information is sent in an encoding mode (INTRAmode) of encoding pictures only within frames without using thedifferences between the frames or the influence of the error graduallywanes to restore the original pictures.

If 1-frame information is lost due to an error, the second frame is notdecoded at all, and, for example, the first frame is directly output. Atthe third frame, a residual error which allows proper decoding only whenit is added to the second frame is added to the first frame. As aconsequence, the third frame is decoded into a picture totally differentfrom the proper picture. Subsequently, residual errors are added towrong pictures. Basically, therefore, the error does not disappear, andproper decoded pictures cannot be played back.

In order to solve this problem, in the prior art, a technique called“refresh” is generally used, in which encoding is performed in the INTRAmode in a predetermined cycle. When encoding is performed in the INTRAmode, since the number of coded bits increases, the quality of a picturewithout any error greatly deteriorates. For this reason, a periodicrefresh method or the like is usually used, which refreshes severalmacroblocks per frame instead of refreshing an entire frame at once. Inthis periodic refresh method, however, although an increase in thenumber of coded bits can be suppressed, a long period of time isrequired to recover a normal state.

Other measures against errors include a measure of using errorcorrection codes. Although this scheme can correct errors that occurrandomly, it has difficulty in coping with errors of several hundredbits that consecutively occur in a burst manner. Even if the scheme cancope with such errors, considerable redundancy occurs.

Techniques have been studied to receive error information and the likeabout a network from a system and adaptively process the errorinformation and the like on the server side. More specifically, such atechnique uses a method of performing re-encoding upon reception oferror information or switching a plurality of files. In this method, theserver needs to have an encoding function and a function of adaptivelyswitching a plurality of files, resulting in extra processing.

As described above, according to the conventional picture encodingtechniques, loss of information due to an error causes a greatdeterioration in picture quality. In addition, a technique such as theperiodic refresh method of reconstructing information lost due to anerror requires a long period of time to achieve error recovery inconsideration of the encoding efficiency. Shortening the time requiredfor recovery will increase the number of encoded bits to result in adeterioration in encoding efficiency.

BRIEF SUMMARY OF THE INVENTION

It is an object of the present invention to provide a picture encodingmethod and apparatus and a picture decoding method and apparatus whichcan quickly recover from an error even if information is lost by theerror, exhibit high encoding efficiency, and need not perform anyre-encoding.

According to a first aspect of the present invention, there is provideda picture encoding method which comprises receiving an input videosignal, encoding the video signal using a reference picture signal togenerate a video code stream, encoding the reference picture signal togenerate a reference picture code stream, and multiplexing the videocode stream with the reference picture code stream to generate an outputcode stream.

According to a second aspect of the present invention, there is provideda picture encoding apparatus comprising a receiving unit configured toreceive an input video signal, a first encoding unit configured toencode the video signal by using a reference picture signal to generatea video code stream, a second encoding unit configured to encode thereference picture signal to generate a reference picture code stream,and a multiplexing unit configured to multiplex the video code streamand the reference picture code stream to generate an output code stream.

According to the third aspect of the present invention, there isprovided a picture decoding method which comprises receiving an inputcode stream containing a video code stream obtained by encoding a videosignal and a reference picture code stream obtained by encoding areference picture signal, decoding the reference picture code streamcontained in the input code stream to generate a first reference picturesignal, and decoding the video code stream contained in the input codestream by selectively using one of a second reference picture signalobtained from a previous picture signal and the first reference picturesignal to generate a playback picture signal.

According to a fourth aspect of the present invention, there is provideda picture decoding apparatus which comprises an input unit configured toreceive an input code stream containing a video code stream obtained byencoding a video signal and a reference picture code stream obtained byencoding a reference picture signal, a first decoding unit configured todecode the reference picture code stream contained in the input codestream to generate a first reference picture signal, and a seconddecoding unit configured to decode the video code stream contained inthe input code stream by selectively using one of a second referencepicture signal obtained from a previous picture signal and the firstreference picture signal to generate a playback picture signal.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

FIG. 1 is a block diagram showing the arrangement of a picture encodingapparatus according to a first embodiment of the present invention;

FIG. 2 is a block diagram showing the arrangement of a picture encodingapparatus to which a header multiplexing unit and control unit areadded, according to the first embodiment;

FIGS. 3A and 3B are views for explaining an example of the datastructure of a picture-associated portion of the output code streamoutput from the picture encoding apparatus according to the firstembodiment;

FIGS. 4A and 4B are views showing an example of the structure of a frameheader code stream of the output code stream output from the pictureencoding apparatus according to the first embodiment;

FIG. 5 is a block diagram showing the arrangement of a picture decodingapparatus according to the first embodiment of the present invention;

FIG. 6 is a view showing a prediction structure for predictive encoding;

FIG. 7 is a view showing error propagation due to predictive encoding;

FIG. 8 is a view for explaining a method of coping with an erroraccording to the first embodiment;

FIG. 9 is a block diagram showing the arrangement of an outputdetermination unit for a reference picture code stream according to thefirst embodiment;

FIG. 10 is a block diagram showing the arrangement of a decodingoperation determination unit for decoding a reference picture codestream according to the first embodiment;

FIG. 11 is a block diagram showing the arrangement of a picture encodingapparatus according to a second embodiment of the present invention;

FIG. 12 is a view showing the data structure of a picture-associatedportion of the output code stream output from the picture encodingapparatus according to the second embodiment;

FIG. 13 is a block diagram showing the arrangement of a picture decodingapparatus according to the second embodiment of the present invention;

FIG. 14 is a block diagram showing the arrangement of a picture encodingapparatus according to a third embodiment of the present invention;

FIG. 15 is a block diagram showing the arrangement of a picture decodingapparatus according to the third embodiment of the present invention;

FIG. 16 is a block diagram showing the arrangement of a picture encodingapparatus according to a fourth embodiment of the present invention; and

FIG. 17 is a block diagram showing the arrangement of a picture decodingapparatus according to the fourth embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION FIRST EMBODIMENT

FIG. 1 shows the basic arrangement of a picture encoding apparatusaccording to the first embodiment of the present invention.

An input video signal 131 is divided into a plurality of predeterminedareas first by an area divider 101 and then subjected to the followingmotion compensation adaptive prediction. A motion compensation adaptivepredictor 111 detects a motion vector 143 between an input picturesignal 132 and a reference picture signal 141 of the previous framewhich is stored in a frame memory 110 and has already been encoded andsubjected to a local decoding. Motion compensation is performed for thereference picture signal 141 by using this motion vector. This generatesa predictive picture signal (the reference picture signal after motioncompensation) 142. The motion compensation adaptive predictor 111selects a suitable prediction mode of the motion compensation predictionmode and the intra encoding (predictive picture signal=0) mode using theinput picture signal 132 for encoding without any change, and outputsthe predictive picture signal 142 corresponding to the selectedprediction mode.

A subtracter 102 subtracts the predictive picture signal 142 from theinput picture signal 132 and outputs a predictive residual error signal133. The predictive residual error signal 133 is subjected to discretecosine transform (DCT) for each block having a given size in a firstdiscrete cosine transformer 103. DCT coefficients 134 obtained by thediscrete cosine transform are quantized by a second quantizer 104. Afirst variable length encoder 105 encodes quantized DCT coefficients 135to obtain a DCT coefficient code stream 136. A multiplexer 106multiplexes the DCT coefficient code stream 136 with a motion vectorcode stream 144 obtained by encoding motion vector information using asecond variable length encoder 112. The resultant data is output as avideo code stream 137.

On the other hand, the DCT coefficient 135 is dequantized by adequantizer 107 and then subjected to an inverse discrete cosinetransform (inverse DCT). An adder 109 adds an output 139 from an inversecosine transformer 108 to the predictive picture signal 142 to generatea local decoded picture signal 140. The local decoded picture signal 140is stored as a reference picture signal in the frame memory 110.

The reference picture signal 141 of the previous frame output from theframe memory 110 is encoded by a reference picture encoding unitcomprising blocks denoted by reference numerals 113 to 115. Morespecifically, the reference picture signal 141 is input to both themotion compensation adaptive predictor 111 and the second discretecosine transformer 113. In the second discrete cosine transformer 113,the reference picture signal 141 is subjected to a discrete cosinetransform (DCT) for each block having a predetermined size. The secondquantizer 114 quantizes transform coefficients 145 obtained by thisoperation. The third variable length encoder 115 encodes the quantizedtransform coefficients. A code stream (to be referred to as a referencepicture code stream hereinafter) 147 obtained by the third variablelength encoder 115 is output as a frame different from the video codestream 137.

FIG. 2 shows an arrangement equivalent to that shown in FIG. 1 whichadditionally includes a header multiplexing unit 117 for adding frameheaders. A control unit 116 manages the overall operation of the pictureencoding apparatus. In this case, in particular, the control unit 116selects, for example, either the encoding mode of outputting only thevideo code stream 137 or the encoding mode of outputting both the videocode stream 137 and the reference picture code stream 147. The controlunit 116 also generates a frame header code stream 148. The headermultiplexing unit 117 generates an output code stream 149 bymultiplexing the video code stream 137, reference picture code stream147, and frame header code stream 148. The output code stream 149 issent out to a transmission system or storage system (not shown).

FIGS. 3A and 3B show the data structure of a picture-associated portionof the output code stream 149 output from the picture encoding apparatusin FIG. 2. In the conventional picture encoding apparatus, a referencepicture signal is not encoded, and only an input video signal isencoded. For this reason, a picture-associated portion of the outputcode stream 149 has a data structure formed from only the video codestream 137 ( . . . Frame #n, Frame #n+1, Frame #n+2, . . . ) of eachframe like that shown in FIG. 3A. In contrast to this, in the pictureencoding apparatus shown in FIG. 2, as shown in FIG. 3B, the referencepicture code stream 147 (Ref-Frame #n+1) in an arbitrary frame, e.g.,the (n+1)-th frame, is inserted before the (n+1)-th frame (Frame #n+1)in the video code stream 137. As a consequence, the frames aremultiplexed to generate the output code stream 149.

In contrast to the case shown in FIG. 3B, a reference picture (147) maybe inserted after the (n+1)-th frame (137).

FIGS. 4A and 4B show examples of the structure of the frame header codestream 148 contained in the output code stream 149. As the modeinformation of the code stream, Pic-type (picture type) information isdefined. As a picture type, R-Picture is separately defined, which isthe frame type of the reference picture code stream 147 unique to thisembodiment, in addition to general I-Picture, P-Picture, and B-Picture.Another method may be used to recognize the reference picture codestream 147. The same effects as described above can be obtained by thismethod.

With regard to a Timestamp indicating the display time of a frame or thelike, it is preferable to describe in an R-picture the Timestamp of aframe using this, i.e., the Timestamp of the next frame. Assume that thereference picture code stream 147 is omitted due to an error or a frameusing the reference picture code stream 147 is omitted due to an error.In this case, such a Timestamp is effective information to identify theassociation between the frame and the reference picture code stream 147.In addition, using the same code stream structure as that of a generalframe eliminates the necessity of a special additional circuit, ageneral circuit can be used.

The use of the scheme of discriminating the modes in accordance withmode information in this manner can implement a recovery function by notonly encoding a reference picture signal used in this embodiment butalso intra-encoding, for example, the target frame itself, which is tobe recovered from an error, and implementing redundancy. A recoveryfunction can be implemented by encoding in advance, in the intra mode, aframe to be subjected to motion compensation adaptive predictiveencoding, and designating only mode information in an R-Picture or thelike. In this case, when it is determined on the transmission side totransmit an R-Picture, there is no need to send the code stream of acorresponding general frame (mainly a P-Picture or B-Picture). Thisembodiment is therefore useful for the effective use of a transmissionpath.

The basic arrangement of a picture decoding apparatus corresponding tothe picture encoding apparatus according to this embodiment will bedescribed with reference to FIG. 5. The output code stream 149 outputfrom the picture encoding apparatus shown in FIG. 2 is input as an inputcode stream to the picture decoding apparatus in FIG. 5 through thetransmission system or storage system. In the picture decodingapparatus, a header demultiplexing unit 200 demultiplexes the input codestream into a video code stream 231 and a reference picture code stream241. A demultiplexer 201 demultiplexes the video code stream 231 into aDCT coefficient code stream 232 and a motion vector code stream 237. TheDCT coefficient code stream 232 is decoded through a variable lengthdecoder 202, dequantizer 203, and inverse discrete cosine transformer204 and reconstructed into a predictive error signal 235. The motionvector code stream 237 is decoded by a variable length decoder 206.Motion vector information 238 reconstructed by this operation is inputto a motion compensation predictor 207.

The reference picture code stream 241 demultiplexed from the input codestream by a header demultiplexing unit (not shown) is transformed into areference picture signal 244 through a variable length encoder 209,dequantizer 210, and inverse discrete cosine transformer 211. This codestream is then stored in a frame memory 208. Using the motion vectorinformation 238, the motion compensation predictor 207 performs motioncompensation for a reference picture signal 239 of the previous frameread out from the frame memory 208 to generate a predictive picturesignal (a reference picture signal after motion compensation) 240. Anadder 205 adds the predictive error signal 235 and the predictivepicture signal 240 to generate a playback picture signal 236. Theplayback picture signal 236 is output to the outside of the apparatusand stored as a reference picture signal in the frame memory 208.

In this embodiment, the picture encoding apparatus sends out theinformation of a reference picture as a reference picture code stream tothe transmission system or storage system independently of a video codestream. The picture decoding apparatus then decodes the referencepicture code stream to reconstruct the information of the referencepicture. This makes it possible to properly cope with the occurrence ofan error. As described above, according to this embodiment, the picturerecovery ability upon occurrence of an error can be improved.

This effect will be further described below. Consider, for example,video encoding operation using a prediction like that shown in FIG. 6.In this case, when an error occurs on the way as shown in FIG. 7, theerror propagates to the subsequent frames using the prediction. For thisreason, such encoding operation generally uses a periodic refresh methodof periodically inserting a frame which can recover from an error, if itoccurs, i.e., an intra-encoded frame (I-Picture). This however degradesthe encoding efficiency.

In contrast to this, according to this embodiment, in the picturedecoding apparatus shown in FIG. 5, the reference picture signalobtained by decoding a reference picture code stream is separately heldas the data of a reference picture signal used upon occurrence of anerror in the frame memory 208, as shown in FIG. 8. This data is used todecode a video code stream only when an error occurs. That is, when noerror occurs, a reference picture signal based on predictive encodingwith high encoding efficiency is used. Only when an error occurs, areference picture signal reconstructed by decoding a reference picturecode stream is used to recover from the error. This apparatus may alsobe configured to forcibly replace the contents of the frame memory 208with a reference picture decoded from a reference picture code streamupon occurrence of an error.

In the arrangement of this embodiment, the total number of codesgenerated in the picture encoding apparatus increases by the extent towhich a reference picture signal is separately encoded. This problem canbe solved by outputting a reference picture code stream only whenneeded. For example, mode information indicating a reference picturecode stream is written at the head of a frame as frame type informationindicating the type of the frame. This mode information is analyzed onthe picture encoding apparatus side to determine whether or not tooutput a reference picture code stream. In the normal mode, no referencepicture code stream is output.

FIG. 9 shows the basic arrangement of an output determination unit forsuch a reference picture code stream. This output determination unit isincorporated in, for example, the header multiplexing unit 117 in FIG.2. Referring to FIG. 9, an input code stream 431 is input to anadditional information determination unit 401, which determines whetherthis code stream is information indicating a normal frame or auxiliarilyadded information. In this embodiment, the input code stream 431 is acode stream including the video code stream 137 and reference picturecode stream 147. The additional information determination unit 401determines whether or not this code stream is the reference picture codestream 147 as additional information.

A determination result 432 from the additional information determinationunit 401 is input to an additional information output determination unit403. This unit determines in accordance with state information 433indicating the current state whether or not to output the referencepicture code stream 147. Assume that the state information 433 isinformation indicating whether or not an error is currently occurring.In this case, if an error is occurring, the additional informationoutput determination unit 403 determines to output the reference picturecode stream 147. In the normal state in which no error is occurring, thereference picture code stream 147 determines not to output the referencepicture code stream 147.

A determination result 434 from the additional information outputdetermination unit 403 is transferred to an output unit 402. The outputunit 402 outputs the reference code stream 147, contained in a codestream 435 input through the additional information determination unit401, as an output code stream 436 in accordance with the determinationresult 434 from the additional information output determination unit403. This makes it possible to adaptively output the reference codestream 147, and hence prevents unnecessary information from being outputin the normal state without any error.

In the picture decoding apparatus shown in FIG. 5, the reference picturecode stream 241 may be input as an input code stream even in the normalstate. For example, such cases include a case wherein a local file isplayed back and a case wherein no adaptive transmission is supported onthe transmission side. In such a case, it can be determined on thepicture decoding apparatus side whether or not to decode the referencepicture code stream 241.

FIG. 10 shows the basic arrangement of a decoding operationdetermination unit which decodes a reference picture code stream upondetermining whether or not to decode it. This decoding operationdetermination unit is incorporated in the header demultiplexing unit 200provided on the front stage in FIG. 5. Referring to FIG. 10, an inputcode stream 531 is input to an additional information determination unit501, which determines whether or not the code stream is additionalinformation. In this embodiment, the input code stream 531 is a codestream containing the video code stream 231 and reference picture codestream 241. The additional information determination unit 501 determineswhether or not the input code stream 531 is the reference picture codestream 241. This determination is performed by using, for example, modeinformation which is written in frame type information and indicates areference picture code stream.

A determination result 532 from the additional information determinationunit 501 is input to a decoding method determination unit 503 to be usedto determine whether or not to decode the reference picture code stream241. Information indicating whether the current decoding operation islocal decoding or an error has occurred is supplied as state information533 to the decoding method determination unit 503. The decoding methoddetermination unit 503 determines from the determination result 532 fromthe additional information determination unit 501 and the stateinformation 533 whether or not to decode the reference picture codestream 241 contained in a code stream 535 input through the additionalinformation determination unit 501. A decoding unit 502 performsdecoding in accordance with a determination result 534 from the decodingmethod determination unit 503 and outputs a playback signal 536. Withthis operation, in the case of local decoding or the like, the picturedecoding apparatus can be controlled not to decode additionalinformation. In the normal state without any transmission error, forexample, the reference picture code stream 241 is discarded by thedecoding method determination unit 503 without being decoded. Assumethat a frame to he referred to is omitted and a playback picture cannotbe normally decoded because an error has occurred in the transmissionpath. In this case, since a reference picture required to decode theplayback picture is not stored in the frame memory, the referencepicture code stream 241 is decoded to replace the picture stored in theframe memory. This prevents a deterioration in the playback picture dueto mixing of the error. This apparatus can also use a technique ofdecoding a reference code stream and replacing the reference frame withthe resultant data only when an error has occurred. The reception sidecan also be configured to decode an entire reference code stream uponreceiving it regardless of whether or not the reference frame is to bereplaced.

This embodiment has been described on the premise that one referenceframe is used. However, a plurality of reference frames may be used. Inthis case, if all the pictures of a plurality of frames are added, thenumber of coded bits may become excessively large, resulting in lack ofpracticality. For this reason, only a small area (e.g., a macroblock inthis case) of a plurality of reference frames which is to be referred toin motion compensation is selected and output as the reference picturecode stream 241. In this case, a data structure per macroblock replacesthe data structure per frame in FIG. 3B. Using this scheme makes itpossible to avoid an increase in the number of coded bits due toencoding of macroblocks that are not used and to save the number ofcoded bits. In this case, a code stream is written together withinformation indicating that macroblock information output as additionalinformation is a macroblock at a specific position in a specific frame.

SECOND EMBODIMENT

FIG. 11 shows the arrangement of a picture encoding apparatus accordingto the second embodiment of the present invention. In the firstembodiment, a reference picture code stream is output as a framedifferent from a video code stream. In the second embodiment, areference picture code stream is output as additional information forthe frames of a video code stream. The same reference numerals as inFIG. 1 denote the same parts in FIG. 11, and only the differences fromthe picture encoding apparatus according to the first embodiment will bedescribed. In this embodiment, a reference picture code stream 147 isinput to a multiplexer 106 to be multiplexed with a quantized andvariable-length-encoded DCT coefficient 136 and motion vector codestream 144. The resultant data is then output.

With this arrangement, a reference picture signal required to encode anddecode a specific frame of a video code stream is added to the frame.More specifically, as shown in FIG. 12, the reference picture codestream 147 (Ref-Frame #n+1) of the (n+1)-th frame becomes additionalinformation in the same (n+1)-th frame (Frame #n+1) in a video codestream 137.

In the scheme of adding a reference picture code stream to a specificframe on a small area (macroblock) basis as described in the latter partof the first embodiment, the apparatus can use a scheme of multiplexingthe video code stream 137 and the reference picture code stream 147 on amacroblock basis and adding, to the video code stream 137, determinationflag information indicating whether or not the reference picture codestream 147 is added to a specific macroblock.

FIG. 13 shows the arrangement of a picture decoding apparatuscorresponding to the picture encoding apparatus in FIG. 11. The samereference numerals as in FIG. 5 denote the same parts in FIG. 13, andonly the differences from the picture decoding apparatus according tothe first embodiment will be described below. In this embodiment, ademultiplexer 201 demultiplexes a reference picture code stream 241,which is inserted as additional information for a frame of a video codestream 231, from the video code stream 231, independently of a DCTcoefficient code stream 231 and motion vector code stream 237. Thedemultiplexed reference picture code stream 241 is decoded through avariable length decoder 209, dequantizer 210, and inverse discretecosine transformer 211 as in the first embodiment, therebyreconstructing a reference picture signal 244. The reference picturesignal obtained by decoding the reference picture code stream is held asthe data of a reference picture signal used upon occurrence of an errorin a frame memory 208. When an error occurs, this reference picture datais used to decode the video code stream. This apparatus may also use thescheme of forcibly replacing the contents of the frame memory 208 withthe reference picture decoded from a reference picture code stream whenan error occurs.

THIRD EMBODIMENT

FIG. 14 shows the arrangement of a picture encoding apparatus accordingto the third embodiment of the present invention. In the first andsecond embodiments, the reference picture code stream 147 is generatedby encoding the reference picture signal stored in the frame memory 110.In the third embodiment, a reference picture code stream 147 isgenerated by encoding a reference picture signal (predictive picturesignal) after motion compensation. In this motion compensation, areference picture obtained by selecting optimal portions from thereference picture signal stored in a frame memory 110 on a small areabasis (mainly on a macroblock basis) is generally stored in the framememory. For this reason, a reference picture signal 142 after motioncompensation is a signal selected from a reference picture signal 141stored in the frame memory 110 on a macroblock basis.

Referring to FIG. 14, the reference picture signal 142, which hasundergone motion compensation (selected on a macroblock basis), outputfrom a motion compensation adaptive predictor 111 is encoded through adiscrete cosine transformer 113, dequantizer 114, and variable lengthencoder 115. As a consequence, the reference picture code stream 147 isgenerated. The reference picture code stream 147 generated in thismanner is output in a frame different from that of a video code stream137 as in the first embodiment, or output after inserted as additionalinformation in the frame of the video code stream 137 as in the secondembodiment.

FIG. 15 shows the arrangement of a picture decoding apparatus accordingto this embodiment which corresponds to the picture encoding apparatusin FIG. 14. A reference picture code stream 241 after motioncompensation, which is demultiplexed from the input code stream by aheader demultiplexing unit 200, is a code stream obtained by encoding areference picture signal after motion compensation, as described withreference to FIG. 14. This code stream is decoded through a variablelength encoder 209, dequantizer 210, and inverse discrete cosinetransformer 211. As a consequence, a reference picture signal 244 aftermotion compensation is reconstructed. The reconstructed referencepicture signal 244 after motion compensation is input to a motioncompensation predictor 207 instead of a frame memory 208 as in the firstand second embodiments.

For example, when an error occurs, the motion compensation predictor 207can output the reference picture signal 244 after motion compensationwhich is reconstructed in the above manner as a predictive picturesignal 240 instead of input motion vector information 238 and areference picture signal 239 from the frame memory 208.

According to this embodiment, even when a plurality of referencepictures are to be used while some manipulations are applied thereto,the reference picture signal 142 after motion compensation is encoded,which is a predictive picture signal directly subtracted, by asubtracter 102, from an input video signal 132 after area segmentation.The reference signal 142 is used as a signal for restoration uponoccurrence of an error. This makes it possible to solve the aboveproblems.

In many cases, a picture frame subjected to predictive encoding ispredictively encoded while it is selected whether predictive encoding(INTER mode) is performed on a macroblock (small area) basis orintra-frame encoding (INTRA mode) is performed. In this case, sincethere is no predictive picture signal in any intra-frame-encodedmacroblocks, if a reference picture signal is output as one frame, anunnecessary portion may be produced. It is therefore possible to selectand store the reference picture code streams 147 required for decodingoperation on a macroblock basis as well as storing reference picturesignals in the frame memory on a frame basis.

FOURTH EMBODIMENT

The first to third embodiments have exemplified the case wherein videoencoding is performed by a combination of motion compensationprediction, discrete cosine transform, quantization, and variable lengthencoding. However, the present invention is not limited to such anencoding scheme. For example, the present invention can be applied tonext-generation encoding techniques such as wavelet encoding.

FIG. 16 shows the basic arrangement of a picture encoding apparatuswhich is generalized in consideretion of such various picture encodingschemes. Referring to FIG. 16, a picture encoding unit 1901 encodes aninput picture signal (mainly a video signal) 1931 using a referencepicture signal to output a picture code stream 1932. The pictureencoding unit 1901 may use any kind of encoding scheme as long as it isa scheme using a reference picture signal. A reference picture signal1933 used by the picture encoding unit 1901 is encoded by a referencepicture encoding unit 1902 and output as a reference picture code stream1934.

FIG. 17 shows the basic arrangement of a picture decoding apparatuscorresponding to the picture encoding apparatus in FIG. 16. A referencepicture code stream 2032 demultiplexed from an input code stream isdecoded by a reference picture decoding unit 2002. As a consequence, areference picture signal 2033 is reconstructed. A picture code stream2031 demultiplexed from the input code stream is decoded by a picturedecoding unit 2001. In this case, if the reference picture signal 2033reconstructed by the reference picture decoding unit 2002 exists as areference picture signal, this signal can be used as is necessary.

FIFTH EMBODIMENT

In general, when a reference picture is encoded in the INTRA mode, anerror is produced between this reference picture and the originalreference picture due to quantization. For this reason, in encodingoperation, the picture encoded signal obtained by conversion/encodingand quantization is used as a reference picture instead of a referencepicture signal as a predictive signal. By transmitting this signal as anadditional reference picture encoded signal to the decoding apparatusside, a system free from errors due to quantization can be realized.

By using the present invention in combination with a feedbackinformation RTCP implementing RTP (Real-time Transport Protocol) or thelike, the effect of the present invention can be enhanced. This isbecause when error information of a network is sent from the receptionside to the transmission side, the information can be used as acondition for determining whether or not to transmit additionalinformation. If, for example, it is determined from RTCP that an errorhas occurred, the reference picture code stream of the next frame istransmitted to the reception side.

Picture encoding and decoding in the present invention described abovemay be implemented by hardware, or part or all of processing may beimplemented by software using a computer. Such software (computerprogram) may be distributed upon being recorded on a recording mediumsuch as a semiconductor memory or CD-ROM, or can be distributed througha transmission medium such as a radio channel or wire.

As described above, according to the present invention, the recoveryability upon occurrence of an error can be improved without anydeterioration in transmission efficiency. In addition, processing withinan encoding framework and preparing all data at the time of encodingwill eliminate the necessity to perform re-encoding, complicatedprocessing at the time of transmission, or the like. This makes itpossible to construct a simple picture transmission/reception system.

As has been described above, the video encoding and decoding apparatusesaccording to the present invention can be used for a system designed tocompression-encode pictures in a small information amount and transmitor store the resultant data in a videophone, video conference system,portable information terminal, digital video disk system, and digital TVbroadcasting system.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

1. A picture decoding apparatus comprising: an input unit configured toreceive an input code stream containing a video code stream obtained byencoding a video signal and a reference picture code stream obtained byencoding a reference picture signal; a first decoding unit configured todecode the reference picture code stream contained in the input codestream to generate a first reference picture signal; a second decodingunit configured to decode the video code stream contained in the inputcode stream by selectively using one of a second reference picturesignal obtained from a previous picture signal and the first referencepicture signal to generate a playback picture signal; a firstdetermination unit configured to determine whether or not the input codestream is the reference picture code stream; a second determination unitconfigured to, when the first determining means determines that theinput code stream is the reference picture code stream, determinewhether or not to decode the reference picture code stream; and a thirddecoding unit configured to decode the input code stream in accordancewith a determination result of the second determination unit.
 2. Apicture decoding method comprising: receiving an input code streamcontaining a video code stream obtained by encoding a video signal and areference picture code stream obtained by encoding a reference picturesignal; decoding the reference picture code stream contained in theinput code stream to generate a first reference picture signal; decodingthe video code stream contained in the input code stream by selectivelyusing one of a second reference picture signal obtained from a previouspicture signal and the first reference picture signal to generate aplayback picture signal; determining whether or not the input codestream is the reference picture code stream; determining whether or notto decode the reference picture code stream when determining that theinput code stream is the reference picture code stream; and decoding theinput code stream in accordance with a determination result of determinewhether or not to decode the reference picture code stream.
 3. Acomputer readable storage medium storing instructions of a computerprogram which when executed by a computer results in performance ofsteps comprising: receiving an input code stream containing a video codestream obtained by encoding a video signal and a reference picture codestream obtained by encoding a reference picture signal; decoding thereference picture code stream contained in the input code stream togenerate a first reference picture signal; decoding the video codestream contained in the input code stream by selectively using one of asecond reference picture signal obtained from a previous picture signaland the first reference picture signal to generate a playback picturesignal; determining whether or not the input code stream is thereference picture code stream; determining whether or not to decode thereference picture code stream when determining that the input codestream is the reference picture code stream; and decoding the input codestream in accordance with a determination result of determine whether ornot to decode the reference picture code stream.
 4. A computer systemcomprising: means for receiving an input code stream containing a videocode stream obtained by encoding a video signal and a reference picturecode stream obtained by encoding a reference picture signal; means fordecoding the reference picture code stream contained in the input codestream to generate a first reference picture signal; means for decodingthe video code stream contained in the input code stream by selectivelyusing one of a second reference picture signal obtained from a previouspicture signal and the first reference picture signal to generate aplayback picture signal; means for determining whether or not the inputcode stream is the reference picture code stream; means for determinewhether or not to decode the reference picture code stream whendetermining that the input code stream is the reference picture codestream; and means for decoding the input code stream in accordance witha determination result of determine whether or not to decode thereference picture code stream.